Mixing Rates for the Gibbs Sampler over Restricted Boltzmann Machines
نویسنده
چکیده
The mixing rate of a Markov chain (Xt)t=0 is the minimum number of steps before the distribution of Xt is close to its stationary distribution with respect to total variation distance. In this work, we give upper and lower bounds for the mixing rate of the Gibbs sampler over Restricted Boltzmann Machines.
منابع مشابه
Mixing Rates for the Alternating Gibbs Sampler over Restricted Boltzmann Machines and Friends
Alternating Gibbs sampling is a modification of classical Gibbs sampling where several variables are simultaneously sampled from their joint conditional distribution. In this work, we investigate the mixing rate of alternating Gibbs sampling with a particular emphasis on Restricted Boltzmann Machines (RBMs) and variants.
متن کاملBoltzmann machines and energy-based models
We review Boltzmann machines and energy-based models. A Boltzmann machine defines a probability distribution over binary-valued patterns. One can learn parameters of a Boltzmann machine via gradient based approaches in a way that log likelihood of data is increased. The gradient and Laplacian of a Boltzmann machine admit beautiful mathematical representations, although computing them is in gene...
متن کاملLayerwise Systematic Scan: Deep Boltzmann Machines and Beyond
For Markov chain Monte Carlo methods, one of the greatest discrepancies between theory and system is the scan order — while most theoretical development on the mixing time analysis deals with random updates, real-world systems are implemented with systematic scans. We bridge this gap for models that exhibit a bipartite structure, including, most notably, the Restricted/Deep Boltzmann Machine. T...
متن کاملBayesian Mixtures of Bernoulli Distributions
The mixture of Bernoulli distributions [6] is a technique that is frequently used for the modeling of binary random vectors. They differ from (restricted) Boltzmann Machines in that they do not model the marginal distribution over the binary data space X as a product of (conditional) Bernoulli distributions, but as a weighted sum of Bernoulli distributions. Despite the non-identifiability of th...
متن کاملDeep Tempering
Restricted Boltzmann Machines (RBMs) are one of the fundamental building blocks of deep learning. Approximate maximum likelihood training of RBMs typically necessitates sampling from these models. In many training scenarios, computationally efficient Gibbs sampling procedures are crippled by poor mixing. In this work we propose a novel method of sampling from Boltzmann machines that demonstrate...
متن کامل